Skip to content

fix(gateway): treat a lock stamped beyond the drift bound as expired - #1600

Merged
kvinwang merged 2 commits into
nextfrom
fix/gateway-lock-future-stamp
Nov 18, 2026
Merged

kvinwang merged 2 commits into
nextfrom
fix/gateway-lock-future-stamp

Conversation

@kvinwang

@kvinwang kvinwang commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

A renew/rotation lease with started_at in the future (a holder clock since stepped back, or a corrupt value) held the key until wall time caught up. For the shared ACME lock there is no force release, so that blocked rotation, CAA reconciliation and first-use registration.

try_acquire_lock now treats a lease stamped more than MAX_CLOCK_DRIFT_SECS (300 s) ahead as expired, the horizon already used for replicated observations; a peer clock ahead within the bound is still honoured. lease_expiry_does_not_overflow asserted the old behaviour and is replaced.

Also fixes comments and a test name that said acquire_acme_lock waits; it fails immediately.

Merge after #1598 (wavekv 2.2): the takeover write must be stamped above the future-stamped entry it replaces, or peers that admitted that entry keep it. Touches the same function as #1599; rebase after it lands (nonce in the test literal).

@kvinwang
kvinwang force-pushed the fix/gateway-lock-future-stamp branch from 30c588d to ee657dd Compare November 18, 2026 08:06
release_lock could delete a successor's lease. A delete is stamped with
local wall time and wins LWW over any lock stamped earlier:

- The value (started_at seconds, started_by) is not unique: after a
  force release the same node can re-acquire in the same second, and
  the stale holder's release matched and deleted the successor's lock.
  Add a random nonce to the value.
- An absent key (already released or force-released) also fell through
  to the delete, which could beat a successor's lock not yet received.
  Delete only when the current value is ours.
- A holder past its timeout could delete a lease a peer had already
  taken over unseen. A peer takes over only once its clock reads
  started_at + timeout, stamping its lock no earlier, so a delete before
  the holder's clock reaches that point cannot beat it whatever the
  skew. Past it, leave the lease to expire; release now takes the
  timeout, like acquire.

The nonce is a serde-default field of a map-encoded value, so older
gateways ignore it and older records decode with nonce 0; the legacy
fixture now encodes the real two-field shape.

Signed-off-by: Kevin Wang <wy721@qq.com>
A renew or rotation lease whose started_at lies in the future (a holder
clock since stepped back, or a corrupt value near u64::MAX) held the key
until wall time caught up. The per-domain lock has an admin force
release, but the shared ACME lock does not, and holding it blocks
rotation, CAA reconciliation and first-use account registration.

Treat a lease stamped more than MAX_CLOCK_DRIFT_SECS ahead of local time
as expired, the same horizon already applied to replicated observations;
a peer clock ahead within the bound is still honoured.

Also correct comments and a test name that described acquire_acme_lock
as waiting: it fails immediately when the lock is held.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
kvinwang force-pushed the fix/gateway-lock-future-stamp branch from ee657dd to aa82462 Compare November 18, 2026 08:28
@kvinwang
kvinwang enabled auto-merge November 18, 2026 08:28
@kvinwang
kvinwang merged commit dcf0823 into next Nov 18, 2026
21 checks passed
@kvinwang
kvinwang deleted the fix/gateway-lock-future-stamp branch November 18, 2026 08:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant